When Things Get Interesting in the Data Center

RTFM

Long before the advent of cloud computing running a data center was an expensive, individual undertaking that often involved renting physical space in the data centers of other companies. Tim was installing servers for a new startup that was freshly funded to the tune of $25 million and had no idea what he was doing. Armed with a trusty screwdriver he racked and stacked servers until an F5 load balancer remained, which he knew nothing about, but could see where it fit in a rack and, plugged in the power and LAN cables, and returned to the Sunnyvale-based office.

The beta went live and worked great, traffic and users grew and so Tim returned to the data center to plug in more servers. It was on this trip that Tim learned how a PDU worked and that the amount of power and wattage supplied to racks mattered. But still, everything worked fine, so he headed home.

Then one night at about 11 pm, he received an angry phone call from the company founder asking why he couldn’t access the website. The company was losing millions a day so he frantically looked at DNS configuration, tried to SSH into the servers and couldn’t, there was no Google for research, no PagerDuty to alert you and help you track down issues, Tim was ready to start stripping new cables. But it turned out that when he went to plug in more servers, he’d removed one of the main cables from the load balancer and it was serving as a failover, to a non-existent primary.

Lessons learned: Read the manual. Oh, and hire a professional who knows what they’re doing.